Theory of Deep Learning III: Generalization Properties of SGD

In Theory III we characterize with a mix of theory and experiments the consistency and generalization properties of deep convolutional networks trained with Stochastic Gradient Descent in classification tasks. A present perceived puzzle is that deep networks show good predicitve performance when overparametrization relative to the number of training data suggests overfitting. We describe an explanation of these empirical results in terms of the following new results on SGD: 1. SGD concentrates in probability like the classical Langevin equation – on large volume, “flat” minima, selecting flat minimizers which are with very high probability also global minimizers. 2. Minimization by GD or SGD on flat minima can be approximated well by minimization on a linear funcion of the weights suggesting pseudoinverse solutions. 3. Pseudoinverse solutions are known to be intrinsically regularized with a regularization parameter λ which decreases as 1 T where T is the number of iterations. This can qualitatively explain all the generalization properties empirically observed for deep networks. 4. GD and SGD are connected closely to robust optimization. This provides an alternative way to show that GD and SGD perform implicit regularization. These results explain the puzzling findings about fitting randomly labeled data while performing well on natural labeled data. They also explains while overparametrization does not result in overfitting. Quantitative, non-vacuous bounds are still missing, as it has almost always been the case for most practical applications of machine learning. We describe in the appendix an alternative approach that explains more directly, with tools of linear algebra the same qualitative properties and puzzles of generalization in deep polynomial networks. This is version 2. The first version was released on 04/04/2017 at https://dspace.mit.edu/handle/1721.1/107841.